Approximate Dynamic Programming via Sum of Squares Programming

We describe an approximate dynamic programming method for stochastic control problems on infinite state and input spaces. The optimal value function is approximated by a linear combination of basis functions with coefficients as decision variables. By relaxing the Bellman equation to an inequality, one obtains a linear program in the basis coefficients with an infinite set of constraints. We show that a recently introduced method, which obtains convex quadratic value function approximations, can be extended to higher order polynomial approximations via sum of squares programming techniques. An approximate value function can then be computed offline by solving a semidefinite program, without having to sample the infinite constraint. The policy is evaluated online by solving a polynomial optimization problem, which also turns out to be convex in some cases. We experimentally validate the method on an autonomous helicopter testbed using a 10-dimensional helicopter model.Comment: 7 pages, 5 figures. Submitted to the 2013 European Control Conference, Zurich, Switzerlan

Multi-agent autonomous surveillance: A framework based on stochastic reachability and hierarchical task allocation

We develop and implement a framework to address autonomous surveillance problems with a collection of pan-tilt (PT) cameras. Using tools from stochastic reachability with random sets, we formulate the problems of target acquisition, target tracking, and acquisition while tracking as reach-avoid dynamic programs for Markov decision processes (MDPs). It is well known that solution methods for MDP problems based on dynamic programming (DP), implemented by state space gridding, suffer from the curse of dimensionality. This becomes a major limitation when one considers a network of PT cameras. To deal with larger problems we propose a hierarchical task allocation mechanism that allows cameras to calculate reach-avoid objectives independently while achieving tasks collectively. We evaluate the proposed algorithms experimentally on a setup involving industrial PT cameras and mobile robots as targets

The Linear Programming Approach to Reach-Avoid Problems for Markov Decision Processes

One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.ISSN:1076-975